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PURPOSE.  To  provide  security  and  policy  review  on  the  document  at  Tab  1  prior  to  release  to  the  public. 

2.  BACKGROUND. 

Authors:  Westmoreland,  David. 

Title.  Assessment  of  scientific  reasoning  as  an  institutional  outcome, 

^Olfii^avamiah^GA*  ^  prCSentati°n  wifl  be  given  at  the  Scholarship  of  Teaching  &  Learning  Commons,  30  Mar  -  01  April 

3.  DISCUSSION.  N/A 

4.  VIEWS  OF  OTHERS.  N/A 

5nn  u1ETMME^ATI0N-,  Sig"C?°rd  bl°Ck  above  ind,catinS  document  is  suitable  for  public  release.  Suitability  is  based  solely 
n  the  document  being  unclassified,  not  jeopardizing  DoD  interests,  and  accurately  portraying  official  policy. 
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PREVIOUS  EDITION  WILL  BE  USED. 


The  US  Air  Force  Academy  has  established  9  institutional  outcomes,  each  of  which  is 
assessed  by  a  team  of  5  - 1 0  faculty  and  staff  members  with  expertise  in  the  outcome 
domam.  Student  achievement  of  the  "Scientific  Reasoning  and  Principles  of  Science" 
was  assessed  in  the  2012-13  academic  year  by  sampling  203  students  distributed 
across  freshman-to-senior  class  years.  Two  assessment  instruments  were  used'  (a)  an 
in-house  survey  of  student  understanding  of  the  Nature  of  Science,  and  (b)  the  Lawson 
est  of  scientific  reasoning.  Students  showed  statistically  significant  gains  in  both 
scores  between  the  sophomore  and  junior  years.  Student  understanding  of  the  Nature 
of  Science  did  not  differ  by  program  of  study,  but  students  in  the  basic  sciences  and 
engineering  scored  significantly  higher  than  students  in  the  humanities  on  the  scientific 
reasoning  assessment.  Overall,  students  were  weakest  when  answering  questions 
related  to  (a)  proportional  reasoning,  (b)  isolation  of  variables,  and  (c)  if-then 
reasoning.  These  findings  are  being  incorporated  into  a  redesign  of  the  core  curriculum 
to  enhance  continuity  among  science  courses  in  presenting  the  Nature  of  Science,  and 
coordination  among  basic  sciences  course  directors  to  align  efforts  to  teach  scientific 
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The  Need  for  Scientific  Literacy,  Part  II 


Courses  that  are  Explicitly  Designed  to 
Emphasize  Science 

Freshman  Yr 

•  Beh  Sci  110 

•  Chem  100 

•  Physics  110 

Sophomore  Yr 

*  Chem  200 

*  Physics  115 

Junior  Yr 

♦  Beh  Sci  310 

•  Biology  315 

Senior  Yr 
•  Mgt  400 
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Approach  of  the  Assessment  Team 

•  Semi-random  sample  to  assess: 

-  Comprehension  of  the  scientific  method  as  a  way 
of  understanding  the  natural  world  (Instrument  A) 

—  Application  of  scientific  reasoning  to  solve 
questions  (Instrument  B) 


Instrument  A 

(Score  ranges  from  0-12) 


Designed  to  assess  whether  students 
comprehend  the  fundamentals  of 
science  as  a  way  of  thinking: 

*  What  is  the  goal  of  science? 

*  What  are  the  limitations  of 
scientific  inquiry? 

*  How  are  the  principles  of 
empiricism,  skepticism, 
and  rationalism  applied? 

*  What  is  the  distinction 
between  fact,  law,  theory? 
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Instrument  B 

(Score  ranges  from  0-12) 


Designed  to  assess  whether  students 
effectively  apply  scientific  reasoning 
to  solve  problems.  Assesses: 

*  Proportional  reasoning 
(example  ~>) 

*  Correlational  reasoning 

*  Probabilistic  prediction 

*  Isolation  of  variables 

*  Hypothetico-deductive  (If,  Then) 
reasoning 

*  Conservation  of  matter 
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Jan  2013 


•  We  sampled  cadets  taking 
Physics  100,  Chem  200, 
Biology  315,  and  Strategic 
Studies  416 

•  N  =  203  cadets 

•  Data  recorded: 

—  Scores  on  both  instruments 

-  Class  year 

-  Academic  division 

—  Core,  Maj,  Cum  G PA 


Clas^  Yr 


2/26/2016 


What  Happens  as  Cadets  Progress 
Through  the  Curriculum? 


Conceptualization  of  Science  as  a  Scientific  Reasoning/Problem 
Way  of  Knowing  solving 


Does  Academic  Division  Matter? 


Conceptualization  of  Science  as  a 
Way  of  Knowing 


Scientific  Reasoning/Problem 
solving 
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Does  Core  GPA  Correlate? 


Conceptualization  of  Science  as  a  Scientific  Reasoning/Problem 
Way  of  Knowing  (r  =  0.335)  solving  (r  =  0.235) 


Which  Factors  are  Significantly  Related 
to  Assessment  Scores? 

Conceptualization  of  Science  as  a  Scientific  Reasoning/Problem 
Way  of  Knowing  solving 


Class  year  P  =  0.03 

Division 

P  =  0.09 

Core  GPA 

P  =  0.25 

Major  GPA 

P  =  0.69 

Class  Yr*  Division 

P  =  0.21 

Class  year  P  =  0.02 

Division 

P  =  0.29 

Core  GPA 

P  =  0.01 

Major  GPA 

P  =  0.42 

Class  Yr*Division 

P  =  0.14 
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Summary  of  Assessment  Findings 

•  Cadets  show  a  significant  improvement  in 
Conceptualization  of  Science  and  Scientific 
Reasoning  between  3-degree  &  2-degree  years 

•  Conceptualization  of  Science:  Firsties  and  2- 
degrees  score  low,  regardless  of  Division 

Scientific  Reasoning:  Firsties  and  2-degrees  score 
about  the  same  as  reported  in  another  study; 

•  Cadets  in  science-focused  Divisions  tend  to  have 
better  skills  in  scientific  reasoning  and  problem 
solving 


Summary  con't. 

°  Core  GPA  correlates  weakly  with  both 
assessment  scores 
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A  WAY  FORWARD 

Conceptualization  of  Science 

0  Biology  480 

—  Explicit  instruction  on 

CONCEPT  Sti£Mt£  ADDRESSES  CtHTAlN  TYPES  OF 

science  as  a  way  of 

thinking  is  effective 

,  . . .  ..  . ’■  fh«.  ::><* .  .  , 

0  The  Core 

-  Coordinate  the  core 

c OMCEPT  TWO .  SCI  E  NCI  ^  SA5L D  G  N  A  f  EW  f  DA  ME  M  T,U 

science  classes  to 

ASSSJiViPUlJsy  S 

introduce  and  reinforce 
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scientific  thinking 

A  WAY  FORWARD 


Scientific  Reasoning 


•  Our  students  are 
weakest  on 


Tvvflpfy  sua  sias  are  p  ated  in  four  glass  subai,  me  tubes  3  re  sealed 
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tsvs-ed.  tuces  are  plates  as  shewn,  Then  Bib>-  are  enpisMd  lu  red  if^nt  j'cn 
f  ,a  *'iiHrlst  The  rymter  d th#j  31  me  urc&v*r*d  pan  pi  each  tuba  ns  4h,av.n  in 
ttia  drawing:, 


-  Proportional  reasoning  ,  ,  ,  , 

(57.5%  correct)  £  a 

-  Isolation  of  variables  •  £  crbr. 


(47%  correct,  ex:  ->  ) 


—  If,  then  reasoning 
(35%  correct) 


Th.j  jsst  resporKr  u?  ►'(■(socnd  wmps  eicva  to  qr  B--.ay 

fnorrtj: 

a.  rec  figtil  but  ntfl  gravity 
t.  gravity  but  not  rad  iignt 
d  to;h  #*d  :lg*ir  and  gravity 
d.  neither  red  light  siftf  gravity 


